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Abstract 

Given a system which can fail in 1 of n different ways, a fault detection and isolation (FDI) algorithm 
uses sensor data in order to determine which fault is the most likely to have occurred. The effectiveness of 
an FDI algorithm can be quantified by a confusion matrix, which indicates the probability that each fault 
is isolated given that each fault has occurred. Confusion matrices are often generated with simulation 
data, particularly for complex systems. In this paper we perform FDI using sums of squares of sensor 
residuals (SSRs). We assume that the sensor residuals are Gaussian, which gives the SSRs a chi-squared 
distribution. We then generate analytic lower and upper bounds on the confusion matrix elements. This 
allows for the generation of optimal sensor sets without numerical simulations. The confusion matrix 
bounds are verified with simulated aircraft engine data. 


1 Introduction 

Many different methods of fault detection and isolation (FDI) have been proposed. Frequency 
domain methods include monitoring resonances [1] or modes [2], Filter-based methods in- 
clude observers [3], unknown input observers [4], Kalman filters [5], particle filters [6], sliding 
mode observers [7], filters [8], and set membership filters [9]. There are also methods 
based on computer intelligence [10], include fuzzy logic [11], neural networks [12], genetic 
algorithms [13], and expert systems [14]. Other methods include those based on Markov 
models [15], system identification [16], wavelets [17], Bayesian inference [18], control input 
manipulation [19], and the parity space approach [20]. Many other FDI methods have also 
been proposed [21], some of which apply to special types of systems. 

The parity space approach to FDI compares the sensor residual vector to nominal user- 
specified fault vectors, and the closest fault vector is isolated as the most likely fault. If the 
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sensor residual vectors are Gaussian, the parity space approach allows an analytic computa- 
tion of the confusion matrix. The FDI approach that we propose is philosophically similar to 
the parity space approach, but instead of using fault vectors, we use sum-of-squared residuals 
(SSRs) in order to detect and isolate a fault. We do not compare our approach with other 
FDI methods, but our approach is chosen because of its amenability to a new statistical 
method for the calculation of confusion matrix bounds. 

If sensor residuals are Gaussian, the SSRs have a chi-squared distribution [22], This allows 
for the specification of SSR bounds for fault detection which have a known false negative rate 
and false positive rate. We can also also compare the SSRs for each fault type to determine 
which fault is most likely to have occurred, and then find analytic bounds for fault isolation 
probabilities. Our FDI algorithm is new, but the primary contribution of this paper is to 
show how confusion matrix element bounds can be derived analytically. The FDI algorithm 
that we propose is fairly simple, but the confusion matrix analysis that we develop is novel 
and its ideas may be adaptable to other FDI algorithms. 

Our approach is to first specify the magnitude of each fault that we want to detect, along 
with a target false positive rate (FPR). For each fault we then find the sensor set that gives 
the largest true positive rate (TPR) for the given FPR. Then we use statistical approaches 
to find confusion matrix bounds. The confusion matrix bounds are the outputs of this 
process. We cannot specify desired confusion matrix bounds ahead of time; the bounds are 
the dependent variables of the sensor selection process. 

The goal of this paper is threefold. Our first goal is to present our SSR-based FDI 
algorithm, which we do in Section 2. Our second goal is to derive confusion matrix bounds, 
which we do in Section 3. Our third goal is to confirm the theory with simulation results, 
which we do in Section 4 using an aircraft turbofan engine model. Section 5 presents some 
discussion and conclusions. 

2 A Simple SSR-Based FDI Algorithm 

This section presents an overview of our simple SSR-based FDI algorithm, gives an overview 
of confusion matrix theory, and provides some fundamental lemmas that are used later. We 
consider FDI of a static, linear system. Sensor residuals are computed at each measurement 
time to perform fault detection, and the sum of the squares of the sensor residuals (SSR) are 
used to perform fault isolation. If the sensor residuals are Gaussian, then the SSRs have a 
chi-squared distribution, which allows the formulation of analytic bounds on the confusion 
matrix elements as discussed in Sections 3. 1-3. 3. 

The residual of the ?’tli sensor is denoted as yi and is a measurement of the difference 
between the sensor output and its nominal output in the no-fault case. In the no-fault case, 
Hi is zero mean. In the fault case, the mean of y l is /q. In either case, the standard deviation 
of yi is <jj. The mean /q depends on which fault occurs, but for simplicity of notation we do 
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( 1 ) 


not indicate that dependence in onr notation. An SSR is given as 

k 


s = ^(Vi/vi) 2 


i = 1 


where k is the number of sensors used in this particular SSR. If each r/i is a zero-mean 
Gaussian random variable, then S' is a random variable with a chi-squared distribution [22], 
Its probability density function (pdf), and cumulative distribution function (cdf), are 


where r(-) is the gamma function and y(-) is the lower incomplete gamma function. 


Note that fault isolation is a different issue than fault detection. Detection means that 
Si > T t for fault detection algorithm i. However, it may be that S) > T) for more than one 
value of i. In that case multiple faults have been detected and a fault isolation algorithm is 
required to isolate the most likely fault. 

If a fault occurs, then the terms in Equation 1 will not, in general, be zero mean. In 
this case S has a noncentral chi-squared distribution [22], Its pdf and cdf are given as 


where A is related to the means /i* of the sensor residuals yi , and /(•) is a modified Bessel 
function of the first kind. 



( 2 ) 



x > 0 
x < 0 


( 3 ) 



( 4 ) 


( 5 ) 


We will use a user-specified threshold T to detect a fault. 


S > T —*■ fault detected 
S < T — > no fault detected 


( 6 ) 

( 7 ) 





oo 


( 9 ) 


J~o 


A 



( 10 ) 



OO 


( 11 ) 
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We use the notation f x (w) and F x (w ) to denote the pdf and cdf of the random variable 
x evaluated at w. In terms of FDI, w will be equal to the SSR. If the random variable that 
is being used is clear from the surrounding text and mathematics, we shorten the notation 
to f{x) and F(x). We use f(x, k) and F(x, k ) to denote the pdf and cdf of the chi-squared 
distribution, and f(x, k, A) and F(x, k, A) to denote the pdf and cdf of the noncentral chi- 
squared distribution. 

A confusion matrix specifies the likelihood of isolating each fault, and can be used to 
quantify the performance of an FDI algorithm. A typical confusion matrix for FDI is shown 
in Table 1. The rows correspond to fault conditions, and the columns correspond to fault 
isolation results. The element in the ith row and jth column is the probability that fault j 
is isolated when fault i occurs. Ideally the confusion matrix would be an identity matrix, 
which would indicate perfect fault isolation. 

In Table 1, CCR* is the probability that fault % is correctly isolated given that it occurs. 
CNR is the probability that a no-fault condition is correctly indicated given that no fault 
occurs. Mij is the probability that fault i is incorrectly isolated given that fault j occurs. 
Mi o is the probability that fault i is incorrectly isolated given that no fault occurs. M 0l is 
the probability that no fault is isolated given that fault i occurs. 

Table 1: Typical confusion matrix format. The rows correspond to fault conditions, and the columns 

correspond to fault isolation results. 



Fault 1 

Fault 2 

Fault n 

No fault 

Fault 1 

CCRi 

M 2 i 


M n i 

M 0 1 

Fault 2 

M 12 

ccr 2 


M n 2 

Mq2 






Fault n 

Min 

M 2n 


CCR n 

Mo n 

No fault 

M W 

m 20 


M n o 

CNR 


In the remainder of this paper we use the following lemmas to derive results. These lemmas 
can be proven using standard definitions and results derived from probability theory [23, 24], 

Lemma 1 The probability that a realization of the random variable x is greater than a 
realization of the random variable y is given as 


r*00 /‘OO 


P(x >y) = 


f{x,y) dxdy 


( 12 ) 


' J y 


where f(x,y) is the joint pdf of x and y. If x and y are independent, this can be written in 
terms of the marginal pdf ’s as 


P(x > y) = 


' — oo 
poo 


f x (w) dw 


f y ( Z ) dz 


[1 - F x (z)\ fy(z) dz 


(13) 

(14) 
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Lemma 2 If y — T + x, where x is a random variable and T is a constant, then 


(15) 

(16) 


= f x (w - T) 
F v (w) = F x (w - T) 


Lemma 3 If y = T — x, where x is a random variable and T is a constant, then 

f y (w) = f x (T — w) (17) 

F y {w) = 1 - F X (T - w ) (18) 

Lemma 4 If z = min(a;, y), where x and y are independent random variables, then 

fz(w) = f x (w)( 1 - F y (w)) + f y (w)( 1 - F x (w)) (19) 


Lemma 5 If z = min(x, T), where x is a random variable and T is a constant, then 


F z (w) 


/,(«>) 


F x (w) 

w <T 

1 

w >T 

fx(w) 

w <T 

0 

w > T 

(1-F X H)5(0) 

w = T 


( 20 ) 

( 21 ) 


where £(•) is the continuous-time impulse function. 

Lemma 6 If z = ma x(x,y), where x and y are independent random variables, then 

fz(w) = f x {w)F y (w ) + fy(w)F x (w ) (22) 


Lemma 7 If z = ma x(x,T), where x is a random variable and T is a constant, then 


F.W = { 

w >T 
w < T 

(23) 

f fx(w) 

w > T 


fz(w ) = < 0 

w <T 

(24) 

1 F x (w)S( 0) 

w = T 
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3 Confusion Matrix Bounds 


This section derives analytic confusion matrix bounds for our SSR-based FDI algorithm. 
Section 3.1 deals with the no-fault case and derives bounds for the correct no-fault rate 
(CNR), which is the probability that no fault is detected given that no fault occurs. It also 
derives bounds for the FPR, which is the probability that at least one fault is detected given 
that no fault occurred. Finally it derives an upper bound for the no-fault misclassihcation 
rate, which is the probability that a given fault is isolated given that no fault occurred. 
Section 3.2 deals with the fault case and derives bounds for the correction classification 
rate (CCR), which is the probability that a given fault is correctly isolated given that it 
occurred. Section 3.3 also deals with the fault case and derives upper bounds for the fault 
misclassihcation rate, which is the probability that the incorrect fault is isolated given that 
some other fault occurred. Section 3.4 summarizes the bounds and their use in the confusion 
matrix, and Section 3.5 discusses the required computational effort. 

3.1 No-Fault Case 

The true negative rate (TNR) is the probability that S < T given that there are no faults. 

The false positive rate (FPR) is the probability that S > T given that there are no faults. 

If only one SSR is considered, these probabilities are given as 

TNR (T,k) = F(T, k) (25) 

FPR (T,fc) = 1 -F(T,k) (26) 

Figure 1 illustrates TNR and FPR for a chi-squared SSR containing k — 10 sensors and a 
detection threshold T = 25. 

3.1.1 False positive rate: Two fault detection algorithms 

Our FDI algorithm will have n fault detection algorithms running in parallel, where n is the 
number of possible faults. This will allow for the calculation of n SSRs, which will allow for 
their comparisons for fault isolation. 

In this section we derive the FPR if two fault detection algorithms are running in parallel. 
Suppose that algorithm 1 attempts to detect fault 1 using k\ a sensors and threshold 7\, and 
algorithm 2 attempts to detect fault 2 using k 2a sensors and threshold T 2 . We use the 
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Figure 1: Illustration of the chi-squared pdf of an SSR with k = 10 sensors. The true negative rate is the 
area to the left of the user-specified threshold T = 25, and the false positive rate is the area to the right of 

the threshold. 


notation 


Y\ 

= {sensors unique to algorithm 1} 

(27) 

y 2 

= {sensors unique to algorithm 2} 

(28) 

Y c 

= {sensors common to algorithms 1 and 2} 

(29) 

3>i 

= {all sensors used by algorithm 1} 

(30) 


= {Yi,Y c } 

(31) 

3^2 

= {all sensors used by algorithm 2} 

(32) 


= {y 2 ,y c } 

(33) 


Note that some of the sensors used to detect fault 1 might also be used to detect fault 2, 
but some of the sensors used to detect fault 1 might not be used for fault 2 (and vice versa). 
Our algorithm therefore takes into account those sensor residuals that are unique to a given 
fault, and also those residuals that are common between faults. We use the notation 3d* to 
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denote the ith normalized residual of the sensors used in algorithm 1, with similar meanings 
for y 2i , Y u , Y 2i , and Y cl . That is, 


5, = E y ?. 

= E^+E 1 ^ 


(34) 

(35) 


If there are no faults, then the probability that either fault 1 or fault 2 will be detected is 


Fp R = p [(e n 2 , > n) or (e yl > T - 

[(E + E y o > n) or (JT Yi + £ Yl > T, 

[(E y o > n - E y i 2 <) (E y i > - E ^ 


- p 
= p 


P\J2 Y i> mill (Ti-EX^-E y : 


(36) 

(37) 

(38) 

(39) 


Note that an d X) ^ are three independent random variables. If neither Y\, 

Y 2 , nor Y c are empty, we can use Lemmas 1, 3, and 4 to obtain 


FPR 

k\ 


/ OO 

[1 - F(y, k c )] [f(Ti - y , h )F(T 2 - y, k 2 ) + f(T 2 

•OO 

ini, k 2 = \Y 2 \, k c =\Y c \ 


V , k 2 )F{T l - y, ki)] dy( 40) 

(41) 


Note that we are using the notation \Z\ to denote the number of elements in the set Z. If 
Yi is empty (sensor set 1 does not have any unique sensors), but Y 2 and Y c are not empty, 
Equation 39 can be written as 


FPR = p[E^>mm(r 1 ,T J -E^) 

= p [E^ > min ( T i>“) 

where a is dehned by the above equation. Lemma 3 tells us that 


fa(y) = f(T 2 -y,k 2 ) 

F a (y ) = l-F{T 2 -y,k 2 ) 

Equations 42 and 43 can be written as 

FPR = p[Erj>5 
b = min {T \ , a) 


Lemma 5 tells us that 

f fa{y) y < Ti 

Mv) = < o y > Ti 

I (1- F a (y))S(0) y — T\ 


(42) 

(43) 

(44) 

(45) 

(46) 

(47) 


(48) 
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We can use Lemma 1 in conjuction with Equations 42-48 to write 


FPR = 


[1 - F(y, k c )] f b (y) dy 


' — OO 

C 

r 

' — OO 

/■Ti 


fTi 


[1 - F(y, fc c )] f a (y) dy+ [1 - F(y, k c )] [1 - F„(i/)] 5(0) dy 


IT \ 


[1 - F(y, k c ) ] /(T 2 - y, k 2 ) dy+[ 1 - F(T„ (1 - F>(T,)] 

[1 - F(y, fc c )] /(T 2 - y, *a) dy + [1 - F(T 2 , Ar c )] F(T, - 7), fcj) 


(49) 

(50) 

(51) 

(52) 


r-min(Ti,T2) 


[1 - F(y, k c )] f(T 2 - y, k 2 ) dy + [1 - F(T 1? k c )] F{T 2 - T u k 2 ) (53) 

J — OO 

If I 2 is empty (sensor set 2 does not have any unique sensors) , but Y\ and Y c are not empty, 
we can use Lemmas 1, 3, and 5 to obtain 

FPR = P Y£ > min (t, - Y 2i) T 2 )] (54) 

/ min(Ti,T 2 ) 

[1 - F(y, k c )] /(^ - y, h) dy + [1 - F(T 2 , k c )\ F(Ti - T 2 , h) (55) 

•OO 

If Y c is empty (the two sensor sets do not have any common sensors), but Yi and Y 2 are not 
empty, we can use Lemmas 3 and 4 to obtain 


FPR = P 


0 > min (t i ~Y,YIT 2 -Y,Y 2 


(56) 


r 0 


[/(^ - y, h)F(T 2 - y, k 2 ) + f{T 2 - y, k^T, - y, k,)} dy (57) 


3.1.2 False positive rate: More than two fault detection algorithms 

The previous section derived the FPR given that two fault detection algorithms are running 
in parallel. Now suppose that there are n > 2 fault detection algorithms. In this case we 
can write 


FPR = P (['.S' > T t ) or ■ ■ ■ or (S n > T n )] 
= 1 — P [(£) <!)),•••, (S n < T„)] 
= 1 - CNR 


(58) 

(59) 

(60) 


where CNR is the correct no-fault rate. CNR is the probability that all of the SSRs are 
below their detection thresholds given that no fault occurred, and can be written as 

CNR = P [(Si < Ti) , • • • , (S n < T n )} (61) 
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This can be bounded as 


'Y[ TNR (Ti, ki ) < CNR < min TNR {T h k n ) (62) 

i = 1 

The lower bound will be exact if none of the fault detection algorithms have any sensors in 
common, which means that TNR(7), ki) and TNR(T), kj ) are independent for all i ^ j. The 
upper bound will be exact if there is some m such that y m is a superset of W for all i ^ m, 
and TNR(T m , k m ) < TNR(T), ki) for all i ^ m. 

We can use Equations 60 and 62 to obtain bounds for the FPR when n > 2 fault detection 
algorithms are running in parallel. 

n 

1 - min TNR(T), k t ) < FPR < 1 - JJ TNR (T u h) (63) 

2—1 


3.1.3 Fault misclassification rates in the no- fault case 


Given that no fault occurred, the probability that fault i is incorrectly isolated is called the 
misclassification rate, M l0 . In this section we derive upper bounds for this probability. 

Suppose that we have n fault detectors running in parallel. Given that a fault is detected, 
that is, that the SSR of at least one fault detector exceeds its threshold, we propose isolating 
the fault using the following logic. 


p = argmaXp 



(64) 


Two fault detection algorithms 


Suppose that we have only two fault detection algorithms, algorithms 1 and 2. Given that no 
fault occurred, the probability that fault 1 is isolated is called the marginal misclassification 
of fault 1 relative to fault 2, and is given as 


m 10j2 = P[(S 1 >T 1 ),(S 1 -T 1 >S 2 ~T 2 )] 


= p 


> max It, - Yci , E y . « + 


If neither Y x , Y 2 , nor Y c are empty, we can use Lemmas 1, 2, 3, and 6 to obtain 

/ OO 

[1 - F(y, k,)] [F(y + T 2 - T u k 2 )f(T \ - y, k c )+ 

-OO 

f(v + T 2 - T„ k 2 ) (1 - F(Ti - y, k c ))} dy 


(65) 

( 66 ) 


(67) 


In order to find finite integration limits that do not result in an excessive loss of accuracy 
in the computation of M 10 2 , we use our knowledge that f(w, k) = 0 for w < 0, and we find 
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values of z 2 and z c such that f(z 2l k 2 ) < e and f(z c ,k c ) < e for some small user-specified 
threshold e. This gives 

Mio, 2 = | C/ [l-F(r/,fc 1 )][F( 2 / + T 2 -T 1 ,fc 2 )/(T 1 - 2 /,fc c )+ (68) 

f(y + T 2 - T u k 2 ) (1 - F(Tx - 2 /, fc c ))] dy 
L = min(Ti - T 2 , Ti - z c ) (69) 

U = max(T 1 , z 2 + T, — T 2 ) (70) 

If 17 is empty (sensor set 1 does not have any unique sensors) , but Y 2 and Y c are not empty, 

the marginal misclassification rate can be derived as 


Mi 


10,2 


0 > max 


(E Y£ + Ti- t 2 , 71 - E Y ci] 
p (E < u - Ti \ p (e u 2 > n ) 


= F(T 2 -T h k 2 )[l~F(T u k c )} 


(71) 

(72) 

(73) 


If Y 2 is empty (sensor set 2 does not have any unique sensors) , but Y\ and Y c are not empty, 
the marginal misclassification rate can be derived as 


^10,2 — P 


E Y £ > max (r, - E V cl U ~ T d 


(74) 


We can use Lemmas 1, 3, and 7 to write this equation as 


M 


10,2 


[1 - F(y, A*)] [/m - y, k c ) + (1 - F(T, - y, k c ))6(y — T 1 + T 2 )] dy (75) 


= [1 - F(Ti - T 2 , ki)} [1 - F(T 2 , k c )} + [ 1 [1 - F(y, h)] f{T, - y, k c ) dy( 76) 

d T 1 —T 2 

If Y c is empty (the two sensor sets do not have any common sensors), but Yi and Y 2 are not 
empty, the marginal misclassification rate can be derived as 

E Yu > max f E Y i + U - U, T, J 
We can use Lemmas 1, 2, and 7 to write this equation as 

poo 

M 10 , 2 = / [l-F( S /,A: 1 )][/(j/ + r 2 -r 1 ,A: 2 ) + F( ? / + r 2 -T 1 ,A: 2 )(5(j/-r 1 )] dy (78) 


^ 10,2 — P 


(77) 


= [1 - F(T\, k,)] F(T 2 , k 2 ) + / [1 - F(y, fc,)] /(y + T 2 -T u k 2 ) dy (79) 

d T\ 
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In order to find finite integration limits that do not result in an excessive loss of accuracy 
in the computation of Mi 0j 2 , we find z such that f(z, ki) < e for some small user-specihed 
threshold e. This gives 


Given that no fault occurred and there are only two fault detection algorithms, the proba- 
bility that fault 1 is isolated is given by Equation 68, 73, 76, or 80. 

More than two fault detection algorithms 

If we have n > 2 fault detection algorithms, the probability that fault 1 is isolated given 
that no fault occurred is the probability that SSRi is greater than its threshold, and greater 
than all of the other SSRs relative to their thresholds. 


So in order to obtain an upper bound for the misclassihcation rate of fault 1 given that 
no fault occurred, we use Equations 68, 73, 76, or 80 as appropriate to find the marginal 
misclassihcation rates. Then we use Equation 82 to find an upper bound for M w . 

3.2 Correct Fault Classification Rates 

The true positive rate (TPR) is defined as the probability that a fault is correctly detected 
(S > T) given that it occurs. This does not take fault isolation into account. It only 
considers the probability that the SSR is greater than its detection threshold. The false 
negative rate (FNR) is defined as the probability that no fault is detected (S < T ) given 
that a fault occurs. If only one SSR is considered, these probabilities can be written as 


where ki a is the total number of sensors in 37, and X ia is the non-centrality parameter of 5) 
given that fault i occurred. Note that ki a > ki because ki is the number of unique sensors 
in 37 , but ki a is the total number of sensors in 37 - Figure 2 illustrates TPR and FNR for a 
chi-squared SSR containing k — 10 sensors, A = 40, and a detection threshold T = 20. 

Given that some fault occurs, we might isolate the correct fault or we might isolate an 
incorrect fault. The probability of isolating the correct fault is called the correct classification 
rate (CCR). In this section we derive lower and upper bounds for the CCR. 



[1- F(y,k 1 )\f(y + T 2 -T 1 ,k 2 )dy (80) 



(81) 

(82) 


FNR* = F(Ti, k ia , A ia ) 
TPR, = 1 — F(Ti, k^, A™) 


(83) 

(84) 
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Figure 2: Illustration of the noncentral chi-squared pdf of an SSR with k = 10 sensors and A = 40. The 
false negative rate is the area to the left of the user-specified threshold T = 20, and the true positive rate is 

the area to the right of the threshold. 


3.2.1 Lower bounds for the correct classification rate 


Suppose we have only two fault detectors, algorithms 1 and 2 , and fault 1 occurs. Consider 
the probability that SSRi is larger than SSR2 relative to their thresholds. We call this the 
marginal detection rate -D12. Note that we are not considering whether or not the SSRs 
exceed their threshold. We are only considering how large the SSRs are relative to their 
thresholds. The marginal detection rate is given as 


D 12 = P[(S 1 -T 1 )>(S 2 -T 2 )} 


( 85 ) 


= P 



-Ti+T 2 


> 



i 


( 86 ) 
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Assuming that neither Y\ nor Y 2 are empty (that is, both sensor sets have some unique 
sensors), we can use Lemmas 1 and 2 to obtain 


oo / /»oo 


D 12 — 


oo \Jy 
roo 


f(x + Ti - 7,, k, : Al l dx j f(y,k 2 )dy 
(1 - F(y + Tl - T- 2 , h, A,)) f(y, k 2 ) dy 


(87) 

( 88 ) 


where 

Ai = Xi V (89) 

If Y\ is empty (sensor set 1 does not have any unique sensors), but Y 2 is not empty, the 
marginal detection rate can be derived as 



= F(T 2 — T x , k 2 ) 


(90) 

(91) 


If Y 2 is empty ( sensor set 2 does not have any unique sensors), but Y\ is not empty, the 
marginal detection rate can be derived as 


£>12 = £ feu 2 . > £i - U j 

(92) 

= l-TIU-U.&j.A,) 

(93) 

Now consider the case where there are n > 2 fault detection algorithms. 
1 occurred, the probability that SSRi is larger than SSRj for all i > 1, 
thresholds, is given as 

Given that fault 
relative to their 

D, = P [(ifd — Ti > S 2 — T 2 ) , • • • , (Si — Ti > S n — T n )] 

(94) 

IV 

H 1=1= 

to 

(95) 

where the inequality arises because the events might not be independent. If none of the fault 
detection algorithms have any sensors in common, then the events are independent and the 


bound is exact. 

Now consider the probability that fault 1 is isolated given that fault 1 occurred. This can 
be written as 


CCRx = P[(S 1 >T 1 ),(S 1 -T 1 >S 2 -T 2 ) r -- ,(S 1 ~T 1 >S n -T n )] (96) 

> TPRiDi (97) 

where the inequality arises because the events might not be independent. Equation 97 gives 
a lower bound for the correct classification rate for fault 1. 
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3.2.2 Upper bounds for the correction classification rate 

Next we find an upper bound for the CCR. To begin, suppose that we have only two fault 
detectors, algorithms 1 and 2. Given that fault 1 occurs, the probability that it is correctly 
isolated is called the marginal CCR. This can be written as 


ccr 12 = r [r,v, - r, > s 2 - t 2 ), (.s', > r, )] 


= p 


X ; Y ii > max X n 2 . + T, - r 2 , r, - X 


(98) 

(99) 


Suppose that both sensor sets have some unique sensors, and the sensor sets also have some 
common sensors. That is, neither hi, Y- 2 , nor Y c are empty. Then we can use Lemmas 1, 2, 
3, and 6 to obtain 

/ OO 

[1 - F(y, k u Ai)] [F(y + T 2 - T u fc 2 )/(71 - y, k c , A c )+ (100) 

-oo 

f(y + T 2 - T u k 2 )( 1 - F{T\ - y, k c , A c ))] dy 


where A c is defined analogously to Ai (see Equation 89). In order to limit the integration 
interval while maintaining accuracy in the computation of CCR 12 , we use our knowledge 
that f(w,k ) and f(w,k, A) are both zero for w < 0, and we numerically find values of z 2 
and z c such that f(z 2 , k 2 ) < e and f(z c , k c , A c ) < e for some small user-specified threshold e. 
This gives 

CCR 12 = J [l-F(y,k 1 ,\ 1 )}[F(y + T 2 -T 1 ,k 2 )f(T 1 -y,k c ,\ c )+ (101) 

f{y + T 2 - T u k 2 )( 1 - F(T\ - y, k c , Ac))] dy 
L = min(Ti - T 2 , T x - z c ) (102) 

U = m&x(Ti,z 2 + T 1 -T 2 ) (103) 


If Yi is empty (sensor set 1 does not have any unique sensors) , but Y 2 and Y c are not empty, 
the marginal CCR can be derived as 


CCRi 2 




F(T 2 -T u k 2 ) [1-F(7i,fc c , Ac)] 


(104) 

(105) 

(106) 
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If Y 2 is empty (sensor set 2 does not have any unique sensors) , but Y\ and Y c are not empty, 
the marginal CCR can be derived as 


CCR 12 = P 


= P 


Y Y ii > Ti - t 2 \ , I Y Y i > n ~Yt 
Y, Yli > max \ Tl - Yj Y<d’Ti - T; 


We can use Lemmas 1, 3, and 7 to write this equation as 


CCR 1 2 = 


[l-FfahAi)] [f(Ti-y,k c , A c )+ 


'Ti-T 2 


(1 - F(Ti - y, k c , A c ))S(y - T, + T 2 )\ dy 
= [1 - F(Ti - T 2 , h, AO] [1 - F(T 2 , k c , A c )] + 

[ [^■^F(y,ki,Xi)]f(T 1 -y,k c ,X c )dy 


1 Ti-t 2 


(107) 

(108) 


(109) 

( 110 ) 


If Y c is empty (the two sensor sets do not have any common sensors), but Yi and Y 2 are not 
empty, the marginal CCR can be derived as 


CCR 12 = P 


Y Y ?<>™* W^+n-n.r, 


We can use Lemmas 1, 2, and 7 to write this equation as 


ccr 12 = 


[1 — F(y, ki, A x )] [f(y + T 2 -T ly k 2 )+ 


it 1 


Fiy + ^-T^Kv-Ti)] dy 
= [l-F(T ll k ll X 1 )]F(T 2l k 2 ) + 

/»oo 

/ [l~F(y,k 1 ,X 1 )]f(y + T 2 ^T 1 ,k 2 )dy 


(111) 

( 112 ) 

(113) 


I Ti 


In order to limit the integration interval while maintaining accuracy in the computation of 
CCRi 2 , we find z such that f(z, k 2 ) < e for some small user-specified threshold e. This gives 


CCR 12 = [1 - F(T 1 , k u AO] F(T 2 , k 2 )+ 


r>2+Ti-T 2 


I Ti 


[1 - F{y, k u AO] fty+n-Tt, k 2 ) dy (114) 


The overall CCR for fault 1 is the probability that SSRi is greater than its threshold, and 
greater than all of the other SSRs relative to their thresholds. 

CCRi = P[(5' 1 >T 1 ),(5' 1 -r 1 >5 2 -T 2 ),---,(5' 1 -T 1 >5 n -r n )] (115) 

< minCCRii (116) 
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So in order to obtain an upper bound for the CCR of fault 1, we use Equations 102, 106, 
110, and 114 as appropriate to find the marginal CCRs. Then we use Equation 116 to find 
the upper bound for CCRi. 


3.3 Fault Misclassification Rates 


In this section we derive upper bounds for the probability that a fault is incorrectly isolated. 
If fault 1 occurs, the probability that fault 2 is isolated is called the misclassification rate 
M 2 \. 

First suppose that we have two fault detection algorithms, algorithms 1 and 2. The 
misclassification rate can then be written as 


M'n = P[(S 2 -T 2 > S 1 -T 1 ),(S 2 >T 2 )} 


= P 


E ^ > max ( E - E ^ 

i \ i i 


(117) 

(118) 


where we use the prime symbol on M' 2l to denote that only two detection algorithms are 
in use. Suppose that both sensor sets have some unique sensors, and the sensor sets also 
have some common sensors. That is, neither Y \ , Y 2l nor Y c are empty. We can then use 
Lemmas 1, 2, 3, and 6 to obtain 


ML = 


/ [1 - F(y, k 2 )} [F(y + T 1 - T 2 , h, A 1 )f(T 2 - y, k Cy A c ) + 

J — OO 

f(v + Ti- T 2 ,k 1} A,)(l - F(T 2 - y,k„ A 0 ))] dy 


(119) 


In order to limit the integration interval while maintaining accuracy in the computation of 
M! n , we use our knowledge that f(w, k, A) is zero for w < 0, and we find values of z\ and 
z c such that f(zi, ki, Ai) < e and f{z c , k c , A c ) < e for some small user-specified threshold e. 
This gives 


ML = 

J U [1 - F(y, k 2 )] [F(y + T, - T 2 , k u Xi)f(T 2 - y, k c , A c ) + 

(120) 


f(y + Ti- T 2 , ki, Ai)(l - F(T 2 - y, k c , A c ))] dy 


L = 

min (T 2 - '/'i • T 2 - z c ) 

(121) 

u = 

max(T 2 , zi + T 2 - 7'i) 

(122) 


If Y\ is empty (sensor set 1 does not have any unique sensors) , but Y 2 and Y c are not empty, 
the misclassification rate can be derived as 


M'i = P 


> max \T 2 -T u T 2 -'^Y* 
i \ i 


(123) 
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We can use Lemmas 1, 3, and 7 to write this as 

■OO 

[1 ~F(y,k 2 )} [f(T 2 -y,k c , A c ) + 


M'i = 


Jt 2 -t i 

(1 - F(T 2 - y , k c , A c ))5(y -T 2 + T,)} dy 
= [1 — F{T 2 — Ti, k 2 )} [1 — F(Ti, k c , A c )] + 

[ [l-F(y,k 2 )]f(T 2 -y,k c ,\ c )dy 

Jt 2 -t 1 


(124) 

(125) 


If Y 2 is empty (sensor set 2 does not have any unique sensors) , but Yi and Y c are not empty, 
the misclassihcation rate can be derived as 


M 2 \ = P 


0 > max 


(^YZ + Tz-TuTi-^YZj 

= p (E 1 « < T > - p (E Y i > 

= F(T 1 ~r 2 ,fc 1 ,A 1 )[l-F(T J ,fc,A c )] 


(126) 

(127) 

(128) 


If Y c is empty (the two sensor sets do not have any common sensors), but Y\ and Y 2 are not 
empty, the misclassihcation rate can be derived as 


M'i = P 


Y1 y £ > m£ “ (E Y i + U - r„ r 2 j 


(129) 


We can use Lemmas 1, 2, and 7 to write this equation as 

•OO 

[1 ~F(y,k 2 )] [f(y + T 1 ~T 2 ,k 1 ,\ 1 )+ 


M-21 = 


(130) 


>t 2 


F(y + T 1 -T 2 ,k 1 ,\ 1 )6{y-T 2 )] dy 


= [1 - F(T 2 , k 2 )] F(Tj, k u AO + / [1 - F(y, k 2 )} f(y + T, - T 2 , Aq, AO dy(131) 

Jt 2 

In order to limit the integration interval while maintaining accuracy in the computation of 
M' 21) we find z such that f(z, k±, AO < e for some small user-specihed threshold e. This gives 

pz+T 2 —T\ 

M' 2l = [1 - F{T 2 , k 2 )] F(T U h, AO + / [1 - F(y, k 2 )] f(y + T, - T 2 , AO dy (132) 

Jt 2 

Given that we have n > 2 detection algorithms running in parallel, the misclassihcation 
rate M 21 is bounded from above by M! n . So in order to obtain an upper bound for M 21 , we 
use Equation 121, 125, 128, or 132 as appropriate. This can be repeated for all marginal 
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(133) 


misclassification rates M^. This gives 


Mij = P[(S 2 -T 2 >S 1 -T 1 ),(S 2 -T 2 >S 3 -T 3 ),--- , 
(S 2 ~ T 2 > S n - T n ), (S 2 > T 2 )} 

< M' ip (i = 1 , • • • , n), (j = 1 , • • • , n), (i ± j ) 


(134) 


Finally we consider the probability Mo* that no fault is indicated when fault i occurs. 
This misclassification rate can be bounded from above as 


3.4 Summary of Confusion Matrix Bounds 

Recall the confusion matrix shown in Table 1. The rows correspond to fault conditions, and 
the columns correspond to fault isolation results. The element in the ith row and jth column 
is the probability that fault j is isolated when fault i occurs. The previous sections derived 
the following bounds. 

• CCR ? ; for i G [1, n) is the probability that fault i is correctly isolated given that it occurs, 
and its lower and upper bounds are given in Equations 97 and 116. 

• CNR is the probability that a no-fault condition is correctly indicated given that no 
fault occurs, and its lower and upper bounds are given in Equation 62. 

• Mij for i,j G [l,n] and i ^ j is the probability that fault i is incorrectly isolated given 
that fault j occurs, and its upper bound is given in Equation 134. 

• Mi o for i G [1, n] is the probability that fault i is incorrectly isolated given that no fault 
occurs, and its upper bound is given in Equation 82. 

• Moi for i G [1, 77 .] is the probability that no fault is isolated given that fault i occurs, 
and its upper bound is given in Equation 137. 

3.5 Computational Effort 

Usually confusion matrices are obtained with simulations. In order to derive an experimental 
confusion matrix with N faults, the number of matrix elements that needs to be calculated 
is on the order of N 2 . (For example, if the number of faults doubles, then the computational 
effort quadruples.) Also, the required number of simulations for each matrix element calcu- 
lation is on the order of N. This is because as the number of possible faults increases, the 
number of simulations required to obtain the same statistical accuracy increases in direct 
proportion. Therefore the computational effort required for the experimental determination 
of a confusion matrix is on the order of N 3 . 


M 0i = P (Si < Ti, ■ ■ ■ ,S n < T n ) 


(135) 

(136) 

(137) 


< P(Si<Ti) 

M 0 i < F(Ti,kia, X ia ) 
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The bounds derived in this paper also require computational effort that is on the order of 
N 3 . This is because each of the bounds summarized in Section 3.4 required computational 
effort on the order of N, and the number of matrix elements is on the order of N 2 . 

4 Simulation Results 

In this section we use simulation results to verify the theoretical bounds of the preceding sec- 
tions. We consider the problem of isolating an aircraft turbofan engine fault which is modeled 
by the NASA Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) [25]. 
There are five possible faults that can occur: fan, low pressure compressor (LPC), high 
pressure compressor (HPC), high pressure turbine (HPT), and low pressure turbine (LPT). 
These five faults entail shifts of both efficiency and flow capacity from nominal values. The 
fault magnitudes that we try to detect are 2.5% for the fan, 20% for the LPC, 2% for the 
HPC, 1.5% for the HPT, and 2% for the LPT. These magnitudes were chosen to give rea- 
sonable fault detection ability. The LPC fault has a very small fault signature, so it is not 
detectable unless its magnitude is relatively large. 

The available sensors and their standard deviations are shown in Table 2. The fault 
influence coefficient matrix shown in Table 3 was generated using C-MAPSS and is based 
on [26] . The numbers in Table 3 are the partial derivatives of the sensor outputs with respect 
to the fault conditions, normalized to the fault percentages discussed above, and normalized 
to one standard deviation of the sensor noise. 

Table 2: Aircraft engine sensors and standard deviations. The temperature sensor standard deviations are 
given in units of degrees Rankine. All other sensor standard deviations are given as a percentage relative to 

their nominal values. 

Symbol Description Standard deviation 


Nc 

Core speed 

0.25% 

P15 

Bypass duct pressure 

0.5% 

P24 

LPC outlet pressure 

0.5% 

Ps30 

HPC outlet pressure 

0.5% 

T24 

LPC outlet temperature 

1 deg 

T30 

HPC outlet temperature 

2 deg 

T48 

HPT outlet temperature 

8 deg 

Wf 

Fuel flow 

0.75% 


In order to decide which sensors to use for detecting each fault, we included one sensor 
at a time in a fault detection algorithm, starting with the sensor with the largest fault 
signature. This gave a total of eight possible sensor sets for each fault detection algorithm. 
We specified a maximum allowable false positive rate of 0.0001, and calculated the smallest 
threshold for each possible sensor set that would give an FPR no greater than 0.0001 (see 
Equation 26 and Figure 1). Given the threshold, we next calculated the true positive rate 
(see Equation 84 and Figure 2). We then chose the sensor set that gave the largest TPR. 
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Table 3: Fault signatures. The rows show the five different fault conditions. The columns show the mean 
sensor value residuals for the given fault conditions, normalized to one standard deviation. 


Sensors 


Faults < 



Nc 

P15 

P24 

Ps30 

T24 

T30 

T48 

Wf 

Fan 

0.80 

2.10 

1.80 

4.05 

0.43 

0.49 

1.21 

3.40 

LPC 

0.00 

0.00 

4.80 

1.20 

3.20 

0.20 

0.80 

0.27 

HPC 

0.72 

0.08 

0.32 

0.64 

0.54 

5.23 

3.08 

1.60 

HPT 

0.96 

0.12 

0.39 

3.27 

0.72 

2.63 

4.40 

2.18 

LPT 

1.20 

0.12 

0.60 

3.56 

0.90 

3.34 

0.03 

2.32 


As an example, consider the fan fault with the normalized fault signatures shown in 
Table 3. The sensors with the largest fault signatures in descending order are Ps30, Wf, 
T30, P15, P24, T48, Nc, and T24. This gives eight potential sensor sets for detecting a fan 
fault: the first potential set uses only sensor Ps30, the second potential set uses Ps30 and 
Wf, and so on. The potential sensor sets along with their detection thresholds and TPRs 
are shown in Table 4. Table 4 shows that using five sensors gives the largest TPR given the 
constraint that FPR < 0.0001. 

Table 4: Potential sensor sets for detecting a fan fault. The thresholds were determined by constraining 
FPR < 0.0001. This table shows that using five sensors gives the largest TPR subject to the FPR 

constraint. 

Sensors Detection threshold Tj True positive rate TPRi 


Ps30 

15.1 

0.563 

Ps30, Wf 

18.4 

0.865 

Ps30, Wf, T30 

21.1 

0.926 

Ps30, Wf, T30, P15 

23.5 

0.949 

Ps30, Wf, T30, P15, P24 

25.7 

0.959 

Ps30, Wf, T30, P15, P24, T48 

27.9 

0.958 

Ps30, Wf, T30, P15, P24, T48, Nc 

29.9 

0.952 

Ps30, Wf, T30, P15, P24, T48, Nc, T24 

31.8 

0.942 


This process described in the previous paragraph was repeated for each fault shown in 
Table 3. The resulting sensor sets are shown in Table 5 and were therefore chosen for FDI 
for each fault type. Note that given an FPR constraint, the detection threshold is a function 
only of the number of sensors in each sensor set; the detection threshold is independent of 
the specific fault signatures. This is illustrated in Figure 1 where it is seen that f(x, k ) is a 
function only of x and k (the number of sensors). 

Finally we used the fault isolation method shown in Equation 64, and we used the equa- 
tions developed in the previous sections to determine theoretical lower and upper bounds for 
the confusion matrix as summarized in Section 3.4. We then ran 100,000 fault simulations 
in order to obtain an experimental confusion matrix. Table 6 shows the theoretical lower 
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bounds of the diagonal elements of the confusion matrix. Lower bounds of the off-diagonal 
elements were not obtained because we are typically more interested in upper bounds of 
off-diagonal elements. Table 7 shows the theoretical upper bounds of the confusion matrix. 
Table 8 shows the experimental confusion matrix. These tables show that the theoretical 
results derived in this paper give reasonably tight bounds to the experimental confusion 
matrix values. 

Recall that we used an FPR of 0.0001 to choose our sensor sets and detection thresholds. 
This means that the first five elements in the last row of Table 7 are guaranteed to be no 
greater than 0.0001. It further means that the element in the lower right corner of Table 6 
is guaranteed to be no greater than 1 — 5(0.0001) = 0.9995. 

Table 5: Sensor sets for fault detection. These sensor sets give the largest TPR for each fault given the 

constraint that FPR < 0.0001. 

Fault i Sensor set i Detection threshold Ti True positive rate TPR^ 


Fan 

Ps30, Wf, T30, P15, P24 

25.7 

0.959 

LPC 

P24, T24 

18.4 

0.943 

HPC 

T30, T48 

18.4 

0.970 

HPT 

T48, Ps30, T30, Wf 

23.5 

0.970 

LPT 

Ps30, T30, Wf 

21.1 

0.844 


Table 6: Lower bounds of diagonal confusion matrix elements. Lower bounds for the off-diagonal elements 
have not been obtained. The rows specify the actual fault condition, and the columns specify the diagnosis. 



Fan LPC HPC HPT LPT No Fault 

Fan 

0.6691 

LPC 

0.9342 

HPC 

0.8692 

HPT 

0.9345 

LPT 

0.6623 

No Fault 

0.9992 


Table 7: 


Upper bounds of confusion matrix elements. The rows specify the actual fault condition, and the 

columns specify the diagnosis. 



Fan 

LPC 

HPC 

HPT 

LPT 

No Fault 

Fan 

0.7761 

0.0000 

0.0001 

0.1115 

0.1899 

0.0408 

LPC 

0.0076 

0.9356 

0.0000 

0.0000 

0.0000 

0.0573 

HPC 

0.0097 

0.0000 

0.8936 

0.0769 

0.0160 

0.0303 

HPT 

0.0015 

0.0000 

0.0270 

0.9445 

0.0014 

0.0300 

LPT 

0.0874 

0.0000 

0.0030 

0.1066 

0.7422 

0.1557 

No Fault 

0.0000 

0.0000 

0.0000 

0.0000 

0.0001 

0.9999 
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Table 8: Experimental confusion matrix. The rows specify the actual fault condition, and the columns 
specify the diagnosis. The numbers are based on 100,000 simulations of each fault. The numbers in each 

row should add up to 1 (within rounding error). 



Fan 

LPC 

HPC 

HPT 

LPT 

No Fault 

Fan 

0.7614 

0.0000 

0.0000 

0.0370 

0.1677 

0.0338 

LPC 

0.0073 

0.9354 

0.0000 

0.0000 

0.0000 

0.0573 

HPC 

0.0051 

0.0000 

0.8875 

0.0713 

0.0074 

0.0288 

HPT 

0.0016 

0.0000 

0.0276 

0.9422 

0.0011 

0.0275 

LPT 

0.0831 

0.0000 

0.0015 

0.0993 

0.6680 

0.1481 

No Fault 

0.0000 

0.0000 

0.0000 

0.0000 

0.0001 

0.9997 


Note that it is possible for an element in the experimental confusion matrix in Table 8 to 
lie outside the bounds shown in Tables 6 and 7. For example, compare the numbers in the 
fourth row and first column in Tables 7 and 8. This is because the numbers in Table 8 are 
experimentally obtained on the basis of a finite number of simulations, and are guaranteed to 
lie within their theoretical bounds only as the number of simulations approaches infinity. In 
fact, that is one of the strengths of the analytic method proposed in this paper. The analytic 
bounds are definite, but simulations are subject to random effects. Also, simulations can 
give misleading conclusions if the simulation has errors. One common simulation error is the 
non-randomness of commonly used random number generators [27]. 

In summary, the user specifies the maximum FPR for each fault, and then finds the sensor 
set that has the largest TPR given the FPR constraint. Analytic confusion matrix bounds 
are then obtained using the theory in this paper. If the results are not satisfactory, the user 
can iterate by changing the maximum FPR constraint. For example, if a TPR is too small, 
then the user will have to increase the FPR constraint. If the confusion matrix bounds of 
fault isolation probabilities are not satisfactory, the user will have to iterate on the FPR 
constraints in order to obtain different confusion matrix bounds. 

5 Conclusion 

This paper has introduced a new fault detection and isolation (FDI) algorithm and derived 
analytical confusion matrix bounds. The FDI algorithm is fairly simple and has not been 
compared with other algorithms. The main contribution of this paper is the generation of 
analytic confusion matrix bounds, and the possibility that our methodology could be adapted 
to other FDI algorithms. Usually confusion matrices are obtained with simulations. Such 
simulations have several potential drawbacks. First, they can be time consuming. Second, 
they can give misleading conclusions if not enough simulations are run to give statistically 
significant results. Third, they can give misleading conclusions if the simulation has errors 
(for example, if the output of the random number generator does not satisfy statistical tests 
for randomness). The theoretical confusion matrix bounds derived in this paper do not 
depend on a random number generator and can be used in place of simulations. 
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Further work in this area could follow several directions. First, the tightness of the 
confusion matrix bounds could be quantified. This paper derives bounds but does not 
guarantee how loose or tight those bounds are. Second, the bounds could be modified to 
be tighter. Third, bounds could be attempted for methods other than the FDI algorithm 
proposed here. The fault isolation method we used isolates the fault that has the largest sum 
of squares of sensor residuals (SSR) relative to its detection threshold. Other fault isolation 
methods could normalize the relative SSR to its standard deviation, or could normalize the 
absolute SSR to its detection threshold. Preliminary efforts have not been successful in 
deriving bounds for these fault isolation methods, but additional effort might bring success. 
All of these FDI methods are static, which means that faults are isolated using measurements 
at a single time. Better fault isolation might be achieved if dynamic system information is 
used. The derivation of theoretical confusion matrix bounds in this case would require 
additional work. 
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